Image capturing and display apparatus and method

ABSTRACT

Provided is a display apparatus and method. The display apparatus may sense light reflected from an object and passed through a display panel, and may control a power of a backlight unit depending on whether the light has passed through the display panel.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the priority benefit of Korean PatentApplication No. 10-2010-0049564, filed on May 27, 2010, in the KoreanIntellectual Property Office, the disclosure of which is incorporatedherein by reference.

BACKGROUND

1. Field

One or more embodiments relate to a display apparatus and method, andmore particularly to a display apparatus and method simultaneously senseincident light to generate an image and displaying an image.

2. Description of the Related Art

A display apparatus may include a display panel, a camera, and abacklight unit. The display apparatus may operate in a capturing modefor acquiring image data, or in a displaying mode for displaying theimage data.

Two examples of such display apparatus have been discussed in an articleby Hirsch et al., “A Thin, Depth-Sensing LCD for 3D Interaction usingLight Fields,” in Proceedings of SIGGRAPH ASIA December 2009, a similardisplay apparatus was set forth. In a first example, the displayapparatus sequentially includes an angle limiter layer, a displaypanel/a coded mask layer, and a diffusing layer. The display panel iscontrolled to generate a coded mask effect of the coded mask layer, anda camera would be positioned behind the diffusing layer. The capturedimage data would thus be obtained from the surface of the diffusinglayer. In this approach, it was necessary to position the camera a meterbehind the diffusing layer to compensate for a lack of an angle-limitingfilm, which the authors of the article indicated was necessary, ratherthan suffering the downside affects of using an angle-limiting film infront of the display.

In an alternative approach, the display device sequentially includes thedisplay panel, a mask, a single large-format sensor, and a backlight.The single large-format sensor detects light that is incident on thesensor after passing through the display panel. The authors indicatedthat it was necessary to have the single large-format sensor be made upof a plane of clear photo-detectors in a clear substrate or membrane, asit was necessary that the most light from the backlight be provided tothe display panel. In these two approaches, the external camera and thesingle large-format sensor both resulted in a low resolution capturingcapabilities, which was not as much of a concern to the authors becausethey were more focused on obtaining sufficient orthographic images forgesture detection, which they acknowledged did not require higherquality images.

The Hirsch et al. article additionally notes that the utilization of adense camera array behind the display panel may not be practical and maybe difficult to develop, as the approach would increase the engineeringcomplexity compared to the use of the single large-frame sensor. Asfurther noted by Hirsch et al., such a different sensor approach wouldintroduce backlight non-uniformity issues due to sensor element blockinglight generated by the backlight. Rather, the authors of Hirsch et al.pointed out the benefits of the transparent photodiode singlelarge-format sensor, with a clear substrate or membrane, wasspecifically used so light from the backlight would not be blocked. Theauthors of Hirsch et al. have also noted that alternative technologiesmay not be available in the near future. The authors further note thattheir approach requires an extended image data capturing period when amodified uniformly redundant array (MURA) coding pattern is used, withthe entire display panel being set to produce the MURA coding patternfor a few milliseconds, producing noticeable flickering. The extendedperiod of time required for the MURA coding patter is required so thesingle large-frame sensor can uniformly capture image data that has beenmodulated by the tiled-MURA code.

Accordingly, a display apparatus including a camera or such a singlelarge-frame sensor may have several limitations in acquiring an image,including not being able to accurately sense incident light while alsooperating the display panel to display images based on light radiatingfrom the backlight.

SUMMARY

According to an aspect of one or more embodiments, there may be provideda display apparatus, including a sensor panel configured to sense afirst incident light, incident to the sensor panel subsequent to passingthrough a display, and to at least partially block transmission of lightthrough the sensor panel, and a controller to selectively control thedisplay to be configured to pass the first incident light toward thesensor panel and to control the display to be configured to pass lightout of the display to display an image, wherein the sensor panelincludes plural distinct portions configured to each forward light fromthe sensor panel towards the display.

According to an aspect of one or more embodiments, there may be provideda display apparatus, including a display panel, a sensor panel includingplural sensor units configured to sense incident light from the displaypanel and plural distinct portions configured to each forward light fromthe sensor panel towards the display panel, and a controller toselectively control opacities of pixels and/or sub-pixels of the displaypanel to produce patterns on the display panel for displaying an imageon the display panel, in a second mode, and to form at least one patternon the display panel controlled to be transparent and an area of thedisplay panel surrounding the at least one pattern controlled to beopaque, to capture an image by the at least one sensor unit, in a firstmode, wherein the plural distinct portions are respectively pluralapertures or plural backlight units, with one or more of the pluralapertures including a diffuser element, and with each of the pluralbacklight units respectively generating and directing light toward thedisplay panel.

According to an aspect of one or more embodiments, there may be provideda display method, including selectively configuring a display to form apattern with at least one transparent pixel or sub-pixel to pass a firstincident light through the display and toward a sensor panel, behind thedisplay, and to form an opaque area that at least surrounds the pattern,when a current mode is a first mode, and configuring the display to passlight from a backlight, behind the display, through the display, whenthe current mode is a second mode, sensing the first incident light,having passed through the display, upon incidence to the sensor panelthat at least partially blocks transmission of light, in a first mode,and displaying an image on the display by passing light in a directionfrom the sensor layer through the display, and repeatedly controllingthe current mode to only one of the first mode and the second mode,including at least a changing of the current mode plural times.

The method may include controlling, at a current time in the first mode,the display to have at least one pattern different than a pattern formedat a previous time when the current mode was the first mode, by changingat least one of a number of first pixels of the display that areselectively controlled to be opaque during the first mode, a number ofsecond pixels of the display that are selectively controlled to betransparent during the first mode, positions of one or more pixels ofthe display that are controlled to be opaque during the first mode, andpositions of one or more pixels of the display that are controlled to betransparent pixels during the first mode, than in the previous time.

The method may further include controlling, at a current time in thefirst mode, the display to have at least one pattern different than apattern formed at a previous time when the current mode was the firstmode, by changing at least one of a number of first sub-pixels of thedisplay that are selectively controlled to be opaque during the firstmode, a number of second sub-pixels of the display that are selectivelycontrolled to be transparent during the first mode, positions of one ormore sub-pixels of the display that are controlled to be opaque duringthe first mode, and positions of one or more sub-pixels of the displaythat are controlled to be transparent pixels during the first mode, thanin the previous time.

The method may further include selectively controlling a combination ofa capture rate, for capturing images from the sensor panel, and adisplay rate, for displaying plural images, to be greater than a maximumrefresh rate of the display.

The method may still further include selectively controlling the displayto only pass the first incident light toward the sensor panel in a firstcycle, of plural fixed time length cycles, and controlling the displayto only pass the light out of the display to display the image in asecond cycle, of the plural fixed time length cycles, wherein the fixedtime is a time it takes for the display to perform one refresh. Themethod may include selectively controlling opacities of pixels and/orsub-pixels of the display to produce patterns on the display fordisplaying the image on the display, in a second mode, and to form atleast one pattern on the display controlled to be transparent and anarea of the display surrounding the at least one pattern controlled tobe opaque, to capture an image by at least one sensor unit of thesensor, in a first mode, wherein the display is controlled differentlyin the first mode and the second mode.

Additionally, the method may include determining a depth of an objectrepresented in the first incident light based upon at least one detectedsize of a blur of the at least one pattern detected by the sensor panel,and the determining may further include determining plural depthmeasurements of the object represented in plural first incident lightseach sensed by the sensor panel based respectively upon at least onedetected size of a blur pattern for each sensed incident light, andgenerating 3D information of the object. Still further, the method mayinclude monitoring a movement by the object over plural captured imagesand respectively generated plural 3D information and determining whetherthe monitored movement and generated plural 3D information indicatesthat a gesture, of plural defined gestures, is being made by themovement of the object, and controlling an operation of a device basedon the determined gesture.

The method may include restoring the captured image by calculating adistance-based impulse response based on the determined depth,performing devolution of the captured image using the distance-basedimpulse response to compensate for refraction and/or geometric effectscaused by the at least one pattern of the display, and encoding therestored captured image.

According to an aspect of one or more embodiments, there may be provideda display method, including selectively configuring a display to form apattern with at least one transparent pixel or sub-pixel to pass a firstincident light through the display and toward a sensor panel, behind thedisplay, and to form an opaque area that at least surrounds the pattern,when a current mode is a first mode, and configuring the display to passlight from plural backlights of the sensor panel through the display,when the current mode is a second mode, sensing the first incidentlight, having passed through the display, upon incidence to the sensorpanel that blocks transmission of light, in a first mode, and displayingan image on the display by passing respectively generated light fromeach backlight from the sensor layer through the display, and repeatedlycontrolling the current mode to be one of the first mode and the secondmode, including at least a changing of the current mode plural times.

Additional aspects and/or advantages of one or more embodiments will beset forth in part in the description which follows and, in part, will beapparent from the description, or may be learned by practice of one ormore embodiments of disclosure. One or more embodiments are inclusive ofsuch additional aspects.

BRIEF DESCRIPTION OF THE DRAWINGS

These and/or other aspects and advantages will become apparent and morereadily appreciated from the following description of embodiments, takenin conjunction with the accompanying drawings of which:

FIGS. 1A and 1B illustrate a display apparatus and method, according toone or more embodiments;

FIGS. 2A-2B, 3A-3B, 4A-4B, and 5A-5B illustrate one or more sensorpanels, such as the sensor panel 107 of FIG. 1A, included in a displayapparatus, according to one or more embodiments;

FIGS. 6A through 6C and FIG. 7 illustrate patterns formed by a displaypanel, such as the display panel 103 of FIG. 1A, in a display apparatus,according to one or more embodiments;

FIGS. 8 through 10 illustrate patterns formed by a display panel, suchas the display panel 103 of FIG. 1A, in a display apparatus, accordingto one or more embodiments; and

FIGS. 11A through 11C illustrate a display device, system, and method,according to one or more embodiments.

DETAILED DESCRIPTION

Reference will now be made in detail to one or more embodiments,illustrated in the accompanying drawings, wherein like referencenumerals refer to like elements throughout. In this regard, embodimentsof the present invention may be embodied in many different forms andshould not be construed as being limited to embodiments set forthherein. Accordingly, embodiments are merely described below, byreferring to the figures, to explain aspects of the present invention.

FIGS. 1A and 1B illustrate a display apparatus and one or morecorresponding image capturing and/or displaying methods, according toone or more embodiments.

Referring to FIG. 1A, the display apparatus 101 may include a displaypanel 103, a sensor panel 107, a backlight unit 113, and a controller115, for example. In an embodiment, the sensor panel may include one ormore sensor units 109 and one or more apertures 111. The leftillustration of FIG. 1A shows a more expanded representation of thedisplay apparatus 101, while the right illustration shows how layers ofthe display apparatus can be manufactured. In an embodiment, themanufacturing of the right illustration of FIG. 1A, of the display panel103, sensor panel 107, and backlight unit 113, may be performed by anyconventional semiconductor or optical/display fabrication methodology.According to one or more embodiments, the below differing arrangementsof the display panel 103, sensor panel 107, sensor units 109, apertures111, backlight units 113, and any potential diffusing layers therein,may equally be manufactured in the inventive arrangements by suchsemiconductor or optical/display panel fabrication methodologies. Asshown in the right illustration of FIG. 1A, in an embodiment, the sensorpanel may be constructed of a substrate that can be etched orphoto-resist washed to form the apertures 111, so there may be fixedareas where light may be transmitted through, e.g., from the lowerbacklight unit 113 toward to display panel 103, and fixed areas wherelight would be blocked. Additional, as only an example, if a diffuserwere incorporated into one or more apertures, this incorporation couldalso be performed at the time the sensor panel 107 is manufactured, suchthat when light passed through the one or more apertures the light wouldbe diffused by the diffuser.

In one or more embodiments, the display panel 103 may be a liquidcrystal display (LCD) panel, though embodiments are not limited to thesame. In an embodiment, the display panel 103 may be any panel that mayselectively provide one or more pinholes and/or masks enabling thepassing of incident light. Here, the display panel 103 may operate in acapturing mode or in a displaying mode based on a predeterminedcriterion of the controller 115. For example, when the display panel 103refreshes at a frequency of 120 Hz, the display panel 103 may switchbetween the capturing mode and the displaying mode every refresh overthe total 120 Hz, e.g., totaling 60 Hz in the capturing mode and 60 Hzin the displaying mode. In an embodiment, the refresh rate of thedisplay panel 103 is the number of times per second in which the displaypanel 103 can display the data it is given. When the display panel 103is an LCD panel, for example, the refresh rate may be the number oftimes the LCD panel can change states per second, e.g., changing a pixelor sub-pixel from an opaque state to a transparent state, or anyavailable gradation there between, which may be representative ofcombined times it takes all underlying control operations to beperformed.

Accordingly, the pinholes and/or masks may be selectively provided bythe display panel 103 without all or some of the pinholes or masks inone cycle, e.g., while radiating light out of the display 103, and thepinholes and/or masks may be selectively provided by the display panelwith all or some of the pinholes or masks in another cycle, e.g.,selectively passing light through the display panel 103 toward thesensor panel 107. It may not be required that the pinholes or masks arecompletely removed during the radiating cycle, and it is not necessarythat all available, or the same, pinholes or masks be maintained inevery capturing cycle. Alternatively, in an embodiment, all pinholes ormasks are removed in the displaying mode.

In the capturing mode, the display panel 103 may pass an incident firstlight, e.g., as reflected from an object, to the sensor panel 107.Specifically, the display panel 103 may be controlled to form one ormore patterns for passing the incident first light by controlling afirst pixel or a first sub-pixel, for example, to be opaque and bycontrolling a second pixel or a second sub-pixel, also as an example,different from the first pixel, to be transparent. For example, thedisplay panel 103 may be controlled to form a pattern 105, including acircular hole pattern, and may pass the incident first light through thepattern 105. Here, the pattern 105 may be formed by the second pixel,and though the second pixel has been described as being only a singlepixel or sub-pixel, one or more embodiments include plural neighboringsecond pixels, e.g., which could be used to control the aperturediameter of the pinhole represented by the pattern 105. Likewise,depending on embodiment and differing uses of the ultimately sensedlight, the pattern 105 would not be limited to having only a circularform, and may be non-circular. In an embodiment, the size of the pattern105 may be controlled to have a same size or number of pixels/sub-pixelsfor different sized display panels and may be selectively modified basedon differing resolutions of each display to maintain the same patternsize.

Accordingly, in one or more embodiments, the display panel 103 may becontrolled to form various types of patterns. As only examples, thedisplay panel 103 may be controlled to form the pattern including atleast one of a circular hole, a polygonal hole, and a modified uniformlyredundant array (MURA) by combining the first pixel and the secondpixel. Here, the pattern 105 indicates an optical meaning of a patternthat may pass light instead of blocking light.

The display panel 103 may be controlled to repeatedly configure theformed pattern based on a number of incident first lights to be passed.For example, when two incident first lights are desired to be passed,the display panel 103 may be controlled to provide two patterns. Whenfour incident first lights are desired to be passed, the display panel103 may be controlled to provide four patterns.

The display panel 103 may be controlled to form various types ofpatterns by changing at least one of a number of opaque first pixels, anumber of transparent second pixels, a position of each opaque firstpixel, and a position of each transparent second pixel.

The display panel 103 may be controlled to form a plurality of patterns,and may also change a number of patterns or a position of each patternat predetermined intervals. For example, the display panel 103 may becontrolled to form the plurality of patterns horizontally or vertically,or in any other organized arrangement. In one or more embodiments, someor all of the patterns or corresponding positions may be randomly choseby the controller 115, and/or some of the patterns or correspondingpositions may vary between capturing cycles.

In the displaying mode, the display panel 103 may be controlled toclearly outwardly display image data using a second light emitted fromthe backlight unit 113, radiating light outward toward the display panel103.

The sensor panel 107 may be disposed after or below the display panel103 to acquire image data by sensing the incident first light, e.g.,reflected from the object, and passed through the display panel 103toward the sensor panel 107.

The sensor panel 107 may include one or more sensor units 109 to senseincident light and apertures 111 to pass light radiating from thebacklight, for example, radiating toward the display panel 103. In oneor more embodiments, the sensor panel 107 may be configured such thatthe sensor units 109 and the apertures 111 form a grid pattern, or maybe configured so the sensor units 109 and the apertures 111 form arepeating pattern, noting that alternative patterns are also available.In one or more embodiments, the placement of the sensor units 109 maynot be evenly distributed over the entire sensor panel 107, and may beof increased density in one or more areas compared to the density of theapertures 111 in the one or more areas. Further, in an embodiment,portions of the sensor panel 107 may be made up of all or primarilyapertures 111 or sensor units 109. For example, in an area that maycorrespond to the level of a user's eyes, there may be an increaseddensity of the sensor units 109, so a remote recipient would have thefeeling that the user is looking directly at the camera, and directly atthe recipient. Alternative reasons for changing the densities areequally available.

The sensor units 109 may be represented by at least one sensor, whichmay sense the incident first light, e.g., reflected from an object, aspassed through the display panel 103. As described above, one or moresensor units 109 may be arranged to sense the respective incident firstlight passed through the pattern 105 formed in the display panel 103.Here, when there exists a plurality of patterns 105, such as pluralholes, one or more sensor units 109 may sense the respective incidentfirst lights passed through the plurality of patterns. The number ofsensor units 109 that may be needed to sense the incident light from onesuch pattern may be calculated, or estimated, and the sensor panel 107configured accordingly. In one or more embodiments, the sensor layer 105may be reconfigurable and not fixed as more or less sensor units 109 andmore or less apertures 111 may be selectively used or available.

When the pattern 105 of the pattern is formed based on a pixel unitincluding red (R), green (G), and blue (B), the sensor unit 109 mayacquire image data corresponding to R, G, and B by sensing the incidentlight that has passed through the pattern 105. When the pattern 105 isformed based on a sub-pixel unit, a pattern formed as an R sub-pixel, apattern formed as a G sub-pixel, and a pattern formed as a B sub-pixelmay be sequentially formed in different cycles, for example, or throughneighboring sub-pixels of different colors. Here, to increase luminanceof the captured image, the number of times incident light from the Gsub-pixel is captured may be greater than the respective number of timesincident light from the R or B sub-pixels, within the example 120available cycles. Accordingly, the sensor unit(s) 109 may acquire imagedata corresponding to R, G, and B by sensing the incident light that haspassed through the patterns of the display panel 103.

In an embodiment, such as when the display panel 103 does not include oris controlled to not apply respective color filters to the differentpixels or sub-pixels, each sensor unit 109 may be configured to includea particular color filter to acquire image data corresponding to acorresponding color of the color filter. For example, one sensor unit109 may include a first sensor unit and a second sensor unit. Each ofthe first sensor unit and the second sensor unit may include adifferently colored unit color filter. Here, there may further be athird sensor unit with still another differently colored unit colorfilter. Also, each of the first sensor unit and the second sensor unitmay include a plurality of color filters having different colors.Additionally, as noted above, the densities and arrangements of thedifferently colored filters and corresponding sensor units 109 is notrequired to be equal or even, and their densities and arrangements mayvary based upon embodiment.

When the sensor unit 109 is configured to have an R color filter, thesensor unit 109 may acquire image data corresponding to the color red bysensing the incident light that has passed through the pattern 105. Whenthe sensor unit 109 is configured to have a B color filter, the sensorunit 109 may acquire image data corresponding to the color green bysensing the incident light that has passed through the pattern 105. Whenthe sensor unit 109 is configured to have a B color filter, the sensorunit 109 may acquire image data corresponding to the color blue bysensing the incident light that has passed through the pattern 105. Inone or more embodiments, the arrangement of different filtering sensorunits 109 on the sensor panel 107 may be according to different Bayerpatterns, for example.

When the backlight unit 113 is disposed behind or below the sensor panel107, the aperture 111 of the sensor panel 107 may pass the radiatingsecond light emitted from the backlight unit 113 towards the displaypanel 103. Here, the aperture 111 may be represented by an opticalmember that may selectively pass light, always pass light, or merelyrepresented by an emptiness in the sensor panel 107, instead of blockingthe light. The optical member may have one or more lens features fordirecting light from the backlight unit 113. The aperture 111 may besimilar to the pattern 105 of the display panel 103, and in one or moreembodiments may be controllable to be transparent for the displayingmode and blocking of light in the capturing mode. In an alternateembodiment, the aperture is a fixed optical member that always permitslight to be passed.

Accordingly, when the backlight unit 113 is disposed behind or below thesensor panel 107, the aperture 111 enables the second light emitted fromthe backlight unit 113 and radiating towards the display panel 103 to betransferred to the display panel 103 without being blocked by the sensorpanel 107. Accordingly, the display panel 103 may clearly display theimage data through the apertures 111 even when the sensor units 109block transmission of one or both of the first incident light and thesecond light. Similar to above, the densities of sensing units 109compared to the densities of the apertures 111 may be different, andtheir arrangements are not required to be fixed, equal, or even. In anembodiment, as only an example, the sensor panel is configured toinclude or use a greater number of apertures 111 than the number ofsensor units 109 so as to increase display resolution, or configured toinclude or use a greater number of sensor units 109 than apertures 111,so as to increase captured image resolution.

According to another embodiment, the sensor panel 107 may furtherinclude, in the aperture 111, a diffuser to prevent the radiating lightfrom the backlight unit 113 from being partially concentrated, or thesensor panel 107 may be configured to include the backlight unit 113,e.g., in the position of the aperture 111, instead of using the aperture111.

The display apparatus 101 may further selectively include a diffuserbetween the display panel 103 and the sensor panel 107. Also, thedisplay apparatus 101 may prevent the incident first light, to be sensedby the sensor unit 109 of the sensor panel 107, from being blocked bysuch a diffuser posited before or above the sensor panel 107 byarranging the sensor panel 107 to incorporate the diffuser with theaperture 111 of the sensor panel 107, or when one or more of thebacklight units 113 are arranged on the sensor panel 107 instead of theaperture 111, the sensor panel 107 may also include a diffusing layeratop or before each backlight unit 113 to diffuse the light radiated byeach backlight unit 113.

Additionally, though the refresh rate of the display panel may be fixedand frame rates for displayed images would decrease proportionally tothe capture rate, in an embodiment, when the pattern 105 isintentionally not changed upon entering the displaying mode, e.g., evenif there may be available image data for display in the same area as thepattern 105 of the display panel 103, if select apertures 111neighboring a sensor unit(s) 109 corresponding to the pattern 105 arecontrollable to be opaque, or neighboring backlighting units 113incorporated into the sensor panel 107 are controlled to not emit theirrespective second lights, during the displaying mode, an image may becaptured from the specific sensor unit(s) 109 at the exact time an imageis being displayed through all other portions of the display panel 103depending on the ambient light effect of neighboring second lights thatare radiating through other apertures 111 or emitted by otherneighboring backlighting units 113. In such an embodiment, the displaymay operate at its full refresh rate and images may be captured at theirfull capture rate.

However, as the ambient effects may make such same-time capturingimpracticable, alternative embodiments may include the display apparatus101, or the display device 1100 of FIG. 11A, having the respectivedisplay panels 103 specifically engineered to reduce the ambient lighteffect, e.g., for particular areas of the display. As an alternative,the display apparatus 101 or display device 1100 of FIG. 11A may have afunctional mode, for example, that would permit additional neighboringapertures to be made opaque or additional backlighting units 113 to becontrolled to not emit their respective second lights to reduce theambient light effect, in which case the controller could also change avertical or horizontal scanning ratio or aspect ratio of the displayedimages to compensate for the loss in viewable space of the display panel103. Herein, the term ‘simultaneous’ with regards to displaying an imageand capturing an image means that image capturing can be performednearly at the same time as image displaying, potentially with eithermode following the other in a directly subsequent cycle. Herein,regarding a capturing of an image and displaying of an image, the term‘same time’ means that the displaying and displaying occur within thesame cycle. When capturing and displaying of respective images occur inat the same the sampling rate and display rate, e.g., frames per second,may sum up to be greater than the actual refresh rate of the displaypanel 103. Accordingly, in an embodiment, the controller 115 may controla capture rate for capturing images using the sensor layer 107 and adisplay rate for displaying images using the display panel 103, at thesame time to be in total greater than the maximum refresh rate of thedisplay panel 103. Further, the controller 115 may selectively performthe capturing mode and displaying mode simultaneously or at the sametime.

The controller 115 may control the display panel 103 to enable theacquiring or capturing of an object image in the capturing mode and thedisplaying of image data in the displaying mode by switching the mode ofthe display panel 103 to/from the capturing mode or the displaying modebased on a predetermined criterion. As noted above, in an embodiment thepredetermined criterion may merely be whether the previous operationmode was the capturing mode or the displaying mode, and to currentlyoperate in a different mode, e.g., in an alternating manner. However, inone or more embodiments, the percentage of the cycles, such as thepercentage of cycles in the 120 available cycles (120 Hz refresh rate)of an example LCD panel, for each of the capturing mode and thedisplaying mode may not be uniform.

In an embodiment, more cycles may be designated for the capturing mode,or more cycles may be designated for the displaying mode. In anembodiment, the number of cycles a same mode operation is maintained mayalso change during the total 120 available cycles, as the number ofcycles may vary between such same mode operations that the other modeoperates. These variances between operation modes may be controlled bythe controller 115, for example, and may be based on different factors,such as the desired image capturing resolution, the desired imagecapturing frame-rate, the desired image displaying resolution, or thedesired image displaying resolution. These desires may be usercontrolled and stored in a memory of the display device, and/or may bebased on external factors, such at the amount of available light forcapturing image, the viewing environment where displayed image is beingviewed, the data transportation capabilities or settings between thedisplay device and another display device, such as in teleconferencing,where the captured image data would be encoded and transmitted, eitherwired or wirelessly, to the other display device.

Here, in an embodiment, when the other display device, as an exampleremote display device, is also a display apparatus 101, the remotedisplay device may also be respectively performing capturing mode anddisplaying mode operations and respectively equally controlling any ofthe above controls on image capturing and/or displaying. In anembodiment, the capturing mode and displaying mode configurations andoperations of an example local display device may control the localcapturing mode and displaying mode configurations and operations basedon the capturing mode and displaying mode configuration and operationsettings of the remote display device. Accordingly, in one or moreembodiments, encoded image data may be coupled with informationidentifying the settings of the local display device. Here, the remoteor local display devices may also decode this information identifyingthe settings of the respective local or remote device. The informationmay further include indications of why the settings were instituted,such as due limitations in processing or display/capturing capability ofeither device, or transmission channel limitations.

When the display panel 103 is controlled to be in the capturing mode,the controller 115 may adjust a size of one or more of the patternsformed on the display panel 103 based on a distance between the displaypanel 103 and the sensor panel 107, or a size of the sensor panel 107.For example, the controller 115 may be capable of being used indifferently sized display devices, with different resolutions, and/orwith different distances between the respective display panel 103 andsensor panel 107, e.g., in an embodiment of the display apparatus 101where the distance can be modified or when the controller is firstdetermining the appropriate size or location for the pattern 105 anddetermines the internal dimensions of the display apparatus 101. One ormore embodiments include a system having plural display devices withrespective capturing mode capabilities through respective sensor panels107 and controllers 115, with each of the configurations of the displaydevices not being required to have the same configuration, such as theavailable different arrangements of the pattern 105, sensor units 109,aperture 111, backlight 113, and diffusing elements. Still further, inan embodiment when there are more than two such display devices and morethan two remotely generated images are displayed on the local displaypanel 103, densities of the sensor units 109 in the local sensor panel107 may be greater in the areas corresponding to the displayed images,compared to densities of the sensor units 109 in other portions of thesensor panel 107. With such an arrangement, both remote viewers may beprovided with the feeling that the local viewer is looking directly ateach of them.

The controller 115 may control a power of the backlight unit 113associated with a light emission towards the display panel 103 based onthe mode of the display panel 103 and thereby control the light emittedfrom the backlight unit 113.

For example, when the display panel 103 is in the capturing mode, thecontroller 115 may power off the backlight unit 113 so that the secondlight may not be emitted from the backlight unit 113 towards the displayunit 103, so the second light does not interfere with the sensing of theincident first light. When the display panel 103 is controlled to be inthe displaying mode, the controller 115 may power on the backlight unit113 so that the second light may be emitted from the backlight unit 113towards the display unit 103. Alternatively, in an arrangement where thebacklight unit 113 is below the sensor panel 107, the aperture 111 maybe configured to be controllable to change opacity from opaque totransparent to selectively block the second light, so the back lightwould not have to cycle on and off.

The controller 115 may control a sensor image restoring unit, includedin the controller 115, as only an example, to interpolate image dataacquired by sensing the incident first light using one or more sensorunits 109 of the display panel 103, and to restore image data lost dueto the inclusion of the aperture 111 in the sensor panel 107, comparedto a configuration where the aperture 111 was replaced by another sensorunit 109 that would have sensed the incident first light incident on thesame position of the sensor panel 107. Accordingly, the controller 115may acquire the image data that may more clearly express thecorresponding object. In an embodiment, different sensor units 109 orgroups of sensor units 109 may be capturing image information fordifferent purposes. For example, a sensor unit 109 or a group of sensorunits 109 may be capturing image data for transmission to a remoteviewing device, while alternate sensor units 109 or groups of sensorunits 109 may be capturing image data for motion detection or depthdetection. The depth detection may be used for encoding the capturedimage data in a 3D graphics data format, for example, of used forcontrolling a focus or image compensation based on depth. The motiondetection may be used with both on-surface (touching the surface of thedisplay device) motion detection and/or off-surface (not touching thesurface of the display device) motion detection, for detecting gesturesof the user, for example, which may be used to control operations of thedisplay device. In one or more embodiments, the controller 115 controlsthe separate detections, the on/off surface motion detection, and thedepth detection. Depending on computational complexity, some detectionprocesses may be delegated to alternate processing elements of thedisplay apparatus 101 or display device 1100, such as shown in FIG. 11A.

Accordingly, one or more embodiments above enable a user to manipulateoperations of the display apparatus 101 through a multi-touch interfacebased at least on the captured image data by the sensor panel 107,either by directly touching the display or through off-surface gesturesor other detectable imaging features that can be inferred or set as acontrol of the display apparatus 101. Control of the display apparatusis available by more then the image detected gestures, such as based ondetected environments surrounding the display apparatus 101 or featuresdetected in the facial image of the viewer, including for facialrecognition for secure use of the display apparatus 101 or secure accessof remote information. The mood of the viewer may also be detected. Oneor more embodiments include an interface technology that enables anatural interaction even though the user is at a distance from a screen,such as proximate sensing and touching, gesture recognition andlong-distance gesture input, e.g., beyond multiple touches availablethrough on-surface input methods, and detection of a camera level. Suchmulti-touch interface detection, off-surface gesture detection,off-surface image feature detection, facial recognition, emotiondetection, and proximate and long-distance gesture detection and inputinterface are implemented in one or more embodiments based onconventional known techniques, though the capturing of the necessaryimage and depth information, and transmission and reception of anyavailable remote image data and capture configurations, for example, canonly be achieved through the arrangements set forth herein.

In previous approaches, it was not possible to have eye contact whenoperating a display device in a video call mode, so with the previousapproaches users would not take advantage of such video call modes sincethe user's would not feel that they were viewing each other's face dueto the non-matching between the screen position and camera position.Similarly, due to the drawbacks of the previous approaches, users ofsuch devices would not feel as if the user was physically with a counterparty at a very remote location using a large screen such as ImmersiveWall and Smart Window, as when the users moved closer to the screen, thefront image of the user would be lost because of the necessary distantlocation of the camera and the moving user. As another example, previousapproaches were unable to simultaneously perform capturing of the imageof the user while permitting the user to also control the display devicethrough a gesture interface, as the camera for the capturing of theimage of the user are typically positioned at an upper end of thedisplay, which makes gesture detection difficult. Similarly, when cameraplacement was more appropriate for gesture detection, the placement ofthe camera would not be appropriate for image capturing of the user.Even with the Hirsch et al. system described above, when a camera waspositioned behind the display panel, the image resolution was inadequatefor image capturing and the frame rate would not be satisfactory foraccurate gesture detection.

However, these drawbacks are all overcome by one or more of the abovedescribed embodiments.

An operation of the display apparatus may be more fully understood byreference to FIG. 1B, which illustrates the operation of the displaypanel 103 and sensor panel 107 in acquiring image data with respect toan object with a pinhole camera, according to one or more embodiments.Here, the pinhole camera represents at least the pattern 105 of FIG. 1A,for example. Discussions below regarding operations of the pinholecamera, such as calculations being performed, may be performed by thecontroller 115 of FIG. 1A, or alternative processing devices, such as inthe display device 1100 of FIG. 11A.

Referring to FIG. 1B, the pinhole camera corresponds to a camera formingan image on a sensor pane by employing a pinhole instead of using lensesto focus the incident light. Accordingly, the pinhole camera may use, asthe pinhole, pattern 105 in the display panel 103, and may use thesensor panel 107 as the sensor plane. When a plurality of patterns areformed on the display panel 103, an effect of a plurality of cameras maybe provided.

Two types of blurs may occur based on the size of the pinhole, e.g.,pattern 105 of the display panel 103. For example, when an aperturediameter of the pinhole is relatively large, a blur by a geometriceffect may occur. When the aperture diameter of the pinhole isrelatively small, a blur by a diffraction effect may occur.

The blur b_(G) by the geometric effect may be expressed by the belowEquation 1, for example.

$\begin{matrix}{b_{G} = \frac{a( {d_{o} + d_{i}} )}{d_{o}}} & {{Equation}\mspace{14mu} 1}\end{matrix}$

In Equation 1, d_(o) denotes a distance between an object and thepinhole, a denotes the aperture diameter, and d_(i) denotes a distancebetween the pinhole and the sensor plane.

With respect to a wavelength λ of light, the blur b_(D) caused by theeffect of optical diffraction may be expressed by the below Equation 2,for example

$\begin{matrix}{b_{D} = \frac{2.44\lambda\; d_{i}}{a}} & {{Equation}\mspace{14mu} 2}\end{matrix}$

In Equation 2, d_(i) denotes the distance between the pinhole and thesensor plane.

A size b of a final blur corresponds to a sum of the two effects andthus, may be expressed by the below Equation 3, for example.

$\begin{matrix}{b = {{b_{G} + b_{D}} = {\frac{a( {d_{o} + d_{i}} )}{d_{o}} + \frac{2.44\lambda\; d_{i}}{a}}}} & {{Equation}\mspace{14mu} 3}\end{matrix}$

When the aperture diameter and the distance between the pinhole and thesensor plane are known, the pinhole camera may calculate the size b ofthe blur and calculate the depth of the object, or distance between theobject and the pinhole, according to the below Equation 4, for example

$\begin{matrix}{d_{o} = \frac{a^{2}d_{i}}{{ab} - {2.44\lambda\; d_{i}} - a^{2}}} & {{Equation}\mspace{14mu} 4}\end{matrix}$

Here, using Equation 4, the distance between the object and a displaypanel, or the depth of the object, may be calculated based upon the twoblurs. Briefly, it is noted that this is different than previousapproaches, such as in the above referenced Hirsch et al. article, wherethe authors use Equation 3 (as Equation (1) in Hirsch et al.) for adifferent purpose to achieve a different answer; the authors usedinformation from the two blurs to determine the angular resolution oftheir system to control the number of orthographic views, rather thanfor obtaining depth information.

The pinhole camera may calculate an inverse impulse response based onthe distance d_(o) between the object and the pinhole. The pinholecamera may further restore distance-based image data by performingde-convolution using the inverse impulse response with respect to theblurred image data acquired by the sensor plane, as shown in the belowEquation 5, for example.I(x,y)= I ′(x,y)*H ⁻¹(x,y)  Equation 5:

In Equation 5, I(x, y) denotes the restored image data, I′(x, y) denotesthe blurred image data, and H⁻¹ (x, y) denotes the inverse pulseresponse.

Accordingly, the pinhole camera may acquire more accurate image databased on determined distance between the object to the pinhole. Usingthe display apparatus 101 of FIG. 1A as an example, the controller 115may accordingly determine the distance between the object and thepattern 105 and then restore the image captured by corresponding one ormore sensor units 109 based on that determined distance.

FIGS. 2A and 2B illustrate a sensor panel 201, such as the sensor panel107 of FIG. 1A, of a display apparatus, according to one or moreembodiments.

Referring to FIG. 2A, the sensor panel 201 may include a sensor unit 203to sense incident light and an aperture 205 to pass light radiating frombelow the sensor panel 201. The sensor panel 201 may be configured suchthat the sensor unit 203 and the aperture 205 are arranged in a gridpattern, or may be configured such that the sensor unit 203 and theaperture 205 are arranged in a repeating pattern. Through this, in anembodiment, the sensor units 203 and the apertures 205 may be uniformlydistributed.

Accordingly, in this embodiment, the sensor panel 201 may uniformlyacquire an incident first light, e.g., reflected from an object, byhaving the sensor units 203 and the apertures 205 configured to bedistributed, and may uniformly transfer, to an upper display panel, asecond light radiating from a backlight unit below the sensor panel 201.

When a pattern included in a pattern of the display panel in thecapturing mode is formed based on a pixel unit of the display panel,including R, G, and B pixel units, the sensor panel 201 may acquireimage data corresponding to R, G, and B by sensing the light passedthrough a pattern 211. Also, when the pattern included in the pattern ofthe display panel is formed based on a sub-pixel unit, a pattern 213formed as an R sub-pixel, a pattern 215 formed as a G sub-pixel, and apattern 217 formed as a B sub-pixel may be sequentially formed.Accordingly, the sensor panel 201 may acquire image data correspondingto R, G, and B by respectively sensing the light passing through thepatterns 213, 215, and 217.

Configurations of the sensor pattern 201 are not limited to those shownin FIG. 2A, and thus, configurations of the sensing units to sense theincident light and apertures to pass the radiating light may vary asshown in the different configurations shown in the left, center, andright sensor panel illustrations of FIG. 2B.

FIGS. 3A and 3B illustrate sensor panels 301 and 307, such as the sensorpanel 107 of FIG. 1A, of a display apparatus, according to one or moreembodiments.

Referring to FIG. 3A, the sensor panel 301 may be configured to includea sensor unit with a color filter to sense incident light that haspassed through a pattern of a display panel, such as the pattern 105 ofdisplay panel 103 in FIG. 1, and to acquire respective image datacorresponding to the color of the color filter. The sensor panel 301 mayinclude a first sensor unit and a second sensor unit, for example. Eachof the first sensor unit and the second sensor unit may also includerespective differently colored unit color filters. Additionally, each ofthe first sensor unit and the second sensor unit may include a pluralityof respective differently colored color filters. The first sensor unitsand the second sensor units may be organized according to any Bayerpattern, for example, and different first sensor units and second sensorunits may be filter one or more different colors through different Bayerpattern.

As one example, as shown in the sensor panel 301 of FIG. 3A, the sensorpanel 301 may be configured to have a first sensor unit 303 as red colorunit color filter and to have a second sensor unit 305 as a blue colorunit color filter. As another example, as shown in the sensor panel 307of FIG. 3A, the sensor panel 307 may be configured to have each of thefirst sensor unit 309 and the second sensor unit 311 include a pluralityof RGB color filters.

The configuration of the sensor panel is not limited to those shown insensor panels 301 and 307 of FIG. 3A, and thus, the sensor panel may beconfigured to have various types of color filters as shown the differentconfigurations shown in the upper left, upper right, lower left andlower right sensor panel illustrations in FIG. 3B.

FIGS. 4A-4B illustrates a sensor panel 401, such as the sensor panel107, of a display apparatus, according to one or more embodiments.

Referring to FIG. 4A, the sensor panel 401 may include a sensor unit 403to sense incident light and a backlight unit 405 to radiate lighttowards a display panel 407. In one or more embodiments, the sensorpanel 401 may be configured such that the sensor unit 403 and thebacklight unit 405 are configured in a grid pattern, or the sensor unit403 and the backlight unit 405 may be configured in a repeating pattern,and thus, in one or more embodiments, the sensor units 403 and thebacklight units 405 may be uniformly distributed.

Accordingly, in such an embodiment, the sensor panel 401 may beconfigured to uniformly acquire incident first light, e.g., reflectedfrom an object, and to uniformly radiate light. Here, the sensor units403 and the backlight units 405 may be arranged so they are evenlydistributed. Accordingly, the sensor panel 401 may uniformly acquireincident light from sensor units 403 and uniformly radiate light fromthe backlight units 405 to the display panel 407.

Since the sensor panel 401 includes the backlight unit 405, the lightradiated from the backlight unit 405 to the display panel 407 may bedirectly emitted towards the display panel 407, instead of being emittedtowards the display panel 407 through the sensor panel 401, e.g., asshown in FIGS. 2A and 2B. FIG. 4B also illustrates the sensor panel 401and the display 407, but demonstrates where there are two pinholes ortwo patterns 105 in the display panel 407, which creates a capturedstereo image by respective sensor units 403.

Accordingly, the sensor panel 401 and the backlight unit 405 may beintegrally formed, and thus, it is possible to reduce the internalvolume of the display apparatus, by removing the previously requiredbacklight unit disposed below the sensor panel 401, thereby producinglight with a slim display apparatus 409. Additionally, with thisconfiguration there is an increase in efficiency of light emission overthe arrangement of FIGS. 2A and 2B, for example, as light from a belowbacklight is not being limited to transmission through only theapertures 111.

FIGS. 5A-5B illustrate a sensor panel 501, such as the sensor panel 107of FIG. 1A, in a display apparatus, according to one or moreembodiments.

Referring to FIG. 5A, the sensor panel 501 may include a sensor unit 503to sense incident light and a diffuser 505 to diffuse light radiatingfrom a backlight unit 509. The sensor panel 501 may be configured tohave the sensor unit 503 and the diffuser 505 arranged in a gridpattern, or may be configure to have the sensor unit 503 and thediffuser 505 arranged in a repeating pattern, as only examples.Accordingly, in an embodiment, the sensor units 503 and the diffusers505 may be uniformly distributed.

In such an embodiment, the sensor panel 501 may uniformly acquireincident first light, e.g., reflected from an object, by having thesensor units 503 and the diffusers 505 distributed evenly, and thus, mayuniformly radiate a second light emitted from the backlight unit 509 tothe display panel 507.

According to an embodiment, the display apparatus 511 using the sensorpanel 501 including the diffuser 505 may directly sense incident light,passed through the display panel 507, using the sensor panel 501, andthereby acquire accurate image data with respect to an object, forexample. FIG. 5B also illustrates the sensor panel 501 and the display507, but demonstrates where there are two pinholes or two patterns 105in the display panel 507, which creates a captured stereo image byrespective sensor units 503.

FIGS. 6A through 6C, and FIG. 7 illustrate plural available patternsthat may be formed in a display panel 601, such as the display panel 103of FIG. 1A, in a display apparatus, according to one or moreembodiments.

Referring to FIG. 6A, the display panel 601 may be controlled to form apattern 603 by controlling a first pixel to be opaque and by controllinga second pixel, different from the first pixel, to be transparent, andmay form a pattern to be included in the pattern 603 so that a light,e.g., reflected from an object, may pass through the pattern 603.

The display panel 601 may be controlled to form the pattern 603 toinclude a circular hole pattern. However, as this is only an example,the display panel 601 may be controlled to form a pattern 605, e.g.,including a polygonal hole, and a pattern 607 including a MURA image.

As shown in FIG. 6B, the display panel 601 may be controlled to form apattern of 11×11 pixels (or sub-pixels), in illustration (i), a patternof 17×17 pixels (or sub-pixels), in illustration (ii), and/or a patternof 23×23 pixels (or sub-pixels), in illustration (iii). As shown in FIG.6C, the display panel 601 may equally be controlled to form a MURApattern of 11×11 pixels (or sub-pixels), in illustration (i), a MURApattern of 17×17 pixels (or sub-pixels), in illustration (ii), and/or aMURA pattern of 23×23 pixels (or sub-pixels), in illustration (iii). Thedisplay panel 601 may be controlled to form both a pattern and a MURApattern, with varying pixel or sub-pixel sizes. The illustration (i) ofFIG. 6B may also represent a pattern using a single pixel or singlesub-pixel.

As shown in FIG. 7, the display panel 601 may form a plurality ofpatterns and may pass incident light, e.g., reflected from an object,using a plurality of patterns. The left illustration of FIG. 7demonstrates the above-referenced stereo image, which may be useful forimage capturing of a viewer, while the center and right illustrations ofFIG. 7 demonstrate patterns that may be useful for gesture capturing ororthographic capturing of a view or object with greaterthree-dimensional data, for example. In one or more embodiments, evenwhen plural patterns are used, the controller 115 of FIG. 1A, forexample, may resolve the images both for depth and use some or all ofthe collected image information for generating a single image or one ormore less images than patterns.

FIG. 8 through FIG. 10 illustrate patterns that may be formed by adisplay panel, such as by the display panel 103 of FIG. 1A, included ina display apparatus, according to one or more embodiments.

Here, the display panel may be controlled to form a plurality ofpatterns, and may adjust the number of patterns and/or the positions ofeach pattern by changing at least one of a number of opaque firstpixels, a number of transparent second pixels, a position of each opaquefirst pixel, and a position of each transparent second pixel, forexample.

Referring to FIG. 8, the display panel may horizontally sequence theplurality of patterns by sequentially forming the plurality of patternsin a first column 801, a second column 803, a third column 805, and afourth column 807, at predetermined intervals, e.g., in different cyclesof the available 120 cycles of an example 120 Hz refresh rate of an LCDscreen.

Referring to FIG. 9, the display panel may be controlled to verticallysequence the plurality of patterns by sequentially forming the pluralityof patterns in a first line 901, in a second line 903, and a third line905, at predetermined intervals, e.g., in different cycles.

Referring to FIG. 10, the display panel may be controlled to alternatelyform the plurality of patterns, e.g., by sequentially changing patternpositions according to an interleaving scheme.

FIG. 11A through 11C illustrate a display device, system, and method,according to one or more embodiments.

Referring to FIG. 11A, the display device 1100 includes a display anduser interface 1101, a controller 1115, a multimedia decoder 1120, amultimedia encoder 1130, a central processing unit (CPU), a memory 1150,transmitter/receiver 1160, speaker 1170, and microphone 1180, forexample. The display and user interface 1101 may operate similarly toany of the above described display panels, sensor panels, sensor units,apertures, backlight units, and diffusing arrangements, with thecontroller 1115 operating as the controller 115 of FIG. 1A, for example.The captured image information may be analyzed by the controller 1115 orthe necessary processing may be handed off to the CPU, such as wheremore extensive processing is required in gesture detection, facialdetection, image resolution or convolution, and the like. The displayand user interface 1101 may include user inputs through on/off-screenmotion detection or gesture detection, as only example. The encoder 1130may encode the captured image data, including performing any 3D graphicsdata encoding based upon the determined depth of one or more objects bythe controller 1115, for example. The encoder may also encode capturedaudio data with the encoded image data, e.g., captured through themicrophone 1180, according to any conventional video/audio encodingschemes, such as any MPEG standard, as only an example.

The encoder may also encode with or separately the encoded video andaudio information and control information, e.g., the above discussed theenvironment or settings for the capturing and/or displaying modes andcorresponding set configurations and arrangements, as only example, usedin the image capturing which may be considered by a remote displaydevice, such as in the system of FIG. 11B having first and seconddisplay devices 1100-1 and 1100-2.

The transmitter/receiver 1160 may then transmit the encoded data to theremote display device, such as the second display device 1000-2 of FIG.11B. The transmitter/receiver 1160 may also receive similarly encodedinformation from the remote display device and forward the same to thedecoder 1120, which decodes the video/audio data and any control datafrom the remote display device. The decoded video/audio information isthen output through the display and any decoded control information isreviewed by the controller to determine whether to change the currentimage capturing settings of capturing mode of the display device 1100.

Referring to FIG. 11B, the system 1110 includes the display device 1100and the remote display device of FIG. 11A, as the first display device1100-1 and second display device 1100-2, in one or more embodiments,each being a display device corresponding to the display device 1100 ofFIG. 11A, e.g., including a display and user interface 1101 and encoderand decoder elements 1125. The network 1190 may be any communicationpath available between the first display device 1000-1 and the seconddisplay device 1100-2, such as an Internet Protocol based network orwireless protocol, or a combination of the same. Both first and seconddisplay devices 1100-1 and 1100-2 do not need to be a display devicecorresponding to the display device 1100 of FIG. 11A, and accordingly,it may not be necessary to communicate between display devices how anytransmitted image data was generated.

Referring to FIG. 11C, an alternative view of the display device 1100 ofFIG. 11A is shown, with the viewer using a video call capability of thedisplay device 1100. Though illustrated in FIG. 11C as a mobile phone orsmart phone, the display devices 1100 of FIGS. 11A and 11B, as the firstor second display devices 1101-1 and 1101-2 in the system 1110, may beany same or different type of display device in the system 1110,including televisions, a personal computer and display as a personalcomputer system, tablet computer devices, mobile phones, PDAs,teleconferencing devices, set-top boxes, etc. The system 1110 of FIG.11B is intended to include any of these devices as either of the firstor second display devices 1101-1 and 1101-2, without requiring the firstor second display devices 1101-1 and 1101-2 to be the same type ofdisplay device.

In one or more embodiments, any apparatus, system, and unit descriptionsherein include one or more hardware devices and/or hardware processingelements/devices, e.g., controllers incorporated into the displayingsystem or with hardware elements. For example, one or more embodimentsinclude a controlling method and/or a controller device controlling thedisplay panel to generate the pinhole apertures during the capturingmode and controlling the display panel to display an image during thedisplaying mode, distinctly from a controller that may control thesensor panel, or diffusing aspects. Each of these controllers mayinclude such hardware processing elements. Additionally, one or moreembodiments may include a configuration similar to that of FIG. 11A,including one or more processing elements in the controller, CPU,display, encoder, and/or decoder hardware portions of the mobile device.Accordingly, in one or more embodiments, any described apparatus,system, and unit may further include one or more desirable memories, andany desired hardware input/output transmission devices, as onlyexamples. Further, the term apparatus should be considered synonymouswith elements of a physical system, not limited to a device, i.e., asingle device at a single location, or enclosure, or limited to alldescribed elements being embodied in single respective element/device orenclosures in all embodiments, but rather, depending on embodiment, isopen to being embodied together or separately in differing devices orenclosures and/or differing locations through differing hardwareelements.

In addition to the above described embodiments, embodiments can also beimplemented through computer readable code/instructions in/on anon-transitory medium, e.g., a computer readable medium, to control atleast one processing element/device, such as a processor, computingdevice, computer, or computer system with peripherals, to implement anyabove described embodiment or aspect of any embodiment. The medium cancorrespond to any defined, measurable, and tangible structure permittingthe storing and/or transmission of the computer readable code.Additionally, one or more embodiments include the at least oneprocessing element or device.

The media may also include, e.g., in combination with the computerreadable code, data files, data structures, and the like. One or moreembodiments of computer-readable media include magnetic media such ashard disks, floppy disks, and magnetic tape; optical media such as CDROM disks and DVDs; magneto-optical media such as optical disks; andhardware devices that are specially configured to store and/or performprogram instructions, such as read-only memory (ROM), random accessmemory (RAM), flash memory, and the at least one processing device,respectively. Computer readable code may include both machine code, suchas produced by a compiler, and files containing higher level code thatmay be executed by the computer using an interpreter, for example. Themedia may also be any defined, measurable, and tangible elements of oneor more distributed networks, so that the computer readable code isstored and/or executed in a distributed fashion. In one or moreembodiments, such distributed networks do not require the computerreadable code to be stored at a same location, e.g., the computerreadable code or portions of the same may be stored remotely, eitherstored remotely at a single location, potentially on a single medium, orstored in a distributed manner, such as in a cloud based manner. Stillfurther, as noted and only as an example, the processing element couldinclude a processor or a computer processor, and processing elements maybe distributed and/or included in a single device. There may be morethan one processing element and/or processing elements with pluraldistinct processing elements, e.g., a processor with plural cores, inwhich case one or more embodiments would include hardware and/or codingto enable single or plural core synchronous or asynchronous operation.

The computer-readable media may also be embodied in at least oneapplication specific integrated circuit (ASIC) or Field ProgrammableGate Array (FPGA), as only examples, which execute (processes like aprocessor) program instructions.

While aspects of the present invention has been particularly shown anddescribed with reference to differing embodiments thereof, it should beunderstood that these embodiments should be considered in a descriptivesense only and not for purposes of limitation. Descriptions of featuresor aspects within each embodiment should typically be considered asavailable for other similar features or aspects in the remainingembodiments. Suitable results may equally be achieved if the describedtechniques are performed in a different order and/or if components in adescribed system, architecture, device, or circuit are combined in adifferent manner and/or replaced or supplemented by other components ortheir equivalents.

Thus, although a few embodiments have been shown and described, withadditional embodiments being equally available, it would be appreciatedby those skilled in the art that changes may be made in theseembodiments without departing from the principles and spirit of theinvention, the scope of which is defined in the claims and theirequivalents.

What is claimed is:
 1. A display apparatus, comprising: a sensor panelconfigured to sense a first incident light, incident to the sensor panelsubsequent to passing through a display, and configured to at leastpartially block transmission of light through the sensor panel; and acontroller configured to selectively control the display to pass thefirst incident light toward the sensor panel, to control the display topass light out of the display to display an image, to determine a depthof an object represented in the first incident light based upon at leastone detected size of a blur of at least one pattern detected by thesensor panel, and to restore a captured image by calculating adistance-based impulse response based on the determined depth byperforming devolution of the captured image using the distance-basedimpulse response to compensate for refraction or geometric effectscaused by the at least one pattern of the display, wherein the sensorpanel comprises first plural distinct portions configured to eachforward light from the sensor panel towards the display and secondplural distinct portions configured to each sense incident lightintensity, wherein the second plural distinct portions are distinct fromthe first plural distinct portions and block transmission of lightthrough the sensor panel toward the display.
 2. The display apparatus ofclaim 1, wherein the controller selectively controls the display to onlypass the first incident light toward the sensor panel in a first cycle,of plural fixed time length cycles, and controls the display to onlypass the light out of the display to display the image in a secondcycle, of the plural fixed time length cycles.
 3. The display apparatusof claim 2, wherein the fixed time is a time it takes for the display toperform one refresh.
 4. The display apparatus of claim 1, wherein, inresponse to the controller selectively controlling plural images to becaptured from plural sensings of respective first incident lights,representing a capture rate, from a pattern of the display controlled bythe controller to not change upon the controller controlling the displayto display the image, the controller controls the capture rate plus adisplay rate, for displaying plural images, to be greater than a maximumrefresh rate of the display.
 5. The display apparatus of claim 1,wherein the controller selectively controls opacities of pixels and/orsub-pixels of the display to produce patterns on the display fordisplaying the image on the display, in a second mode, and to form atleast one pattern on the display controlled to be transparent and anarea of the display surrounding the at least one pattern controlled tobe opaque, to capture an image by at least one sensor unit, of thesecond plural distinct portions, in a first mode, wherein the display iscontrolled by the controller differently in the first mode and thesecond mode.
 6. The display apparatus of claim 1, wherein the controllerdetermines plural depth measurements of the object represented in pluralfirst incident lights each sensed by the sensor panel based respectivelyupon at least one detected size of a blur pattern for each sensedincident light, and generates 3D information of the object.
 7. Thedisplay apparatus of claim 6, wherein the controller controls amonitoring of movement by the object over plural captured images andrespectively generated plural 3D information and determines whether themonitored movement and generated plural 3D information indicates that agesture, of plural defined gestures, is being made by the movement ofthe object.
 8. The display apparatus of claim 1, further comprising: adisplay panel, as the display; and a backlight unit configured to emit asecond light towards the sensor panel, wherein the controllerselectively controls the display panel for passing the first incidentlight toward the sensor panel, in a first mode, and controls the displaypanel for passing the second light through the display panel to displaythe image, in a second mode.
 9. The display apparatus of claim 8,wherein the first plural distinct portions are each configured to alwaysforward the second light from the backlight unit towards the displayunit.
 10. The display apparatus of claim 8, wherein the controllercontrols the backlight unit to emit light in the second mode and to notemit light during the first mode.
 11. The display apparatus of claim 8,wherein the display device is one of a television, a personal computersystem, a tablet computer device, a mobile phone, a PDA,teleconferencing devices or system, or set-top box, with each comprisingat least one processor and memory, an encoder to encode an imagecaptured by the sensing by the sensor panel in the first mode, a decoderto at least decode image data received from an external source, and withthe controller forwarding decoded image data from the decoder to thedisplay panel and controlling the display panel to display the decodedimage data in the second mode.
 12. The display apparatus of claim 11,wherein the controller selectively controls opacities of pixels and/orsub-pixels of the display panel to produce patterns on the display panelfor displaying the image on the display panel, in a second mode, and toform at least one pattern on the display panel controlled to comprise atleast one transparent pixel or sub-pixel and an area of the displaypanel surrounding the at least one pattern controlled to be opaque, tocapture an image by at least one sensor unit of the sensor panel, in afirst mode, wherein the display panel is controlled by the controllerdifferently in the first mode and the second mode.
 13. The displayapparatus of claim 11, wherein the controller controls a monitoring ofmovement of an object, represented in the first incident light, overplural captured images and respectively generated plural 3D informationfor the object and determines whether the monitored movement andgenerated plural 3D information indicates that a gesture, of pluraldefined gestures, is being made by the movement of the object.
 14. Thedisplay apparatus of claim 1, wherein, in response to the controllerselectively controlling plural images to be captured from pluralsensings of respective first incident lights, representing a capturerate, from a pattern of the display controlled by the controller to notchange upon the controller controlling the display to display the image,the controller controls the capture rate plus a display rate, fordisplaying plural images, to be greater than a maximum refresh rate ofthe display.
 15. The display apparatus of claim 1, further comprising: adisplay panel, as the display; and plural backlight units respectivelybeing the first plural distinct portions of the sensor panel, eachconfigured to respectively emit a second light towards the displaypanel, wherein the controller selectively controls the display panel topass the first incident light toward the sensor panel in a first modeand controls the display panel to pass each of the second lights throughthe display panel in a second mode.
 16. The display apparatus of claim15, wherein the controller controls each of the plural backlight unitsto emit light in the second mode and to not emit light during the firstmode.
 17. The display apparatus of claim 15, wherein the display deviceis one of a television, a personal computer system, a tablet computerdevice, a mobile phone, a PDA, teleconferencing devices or system, orset-top box, with each comprising at least one processor and memory, anencoder to encode an image captured by the sensing by the sensor panelin the first mode, a decoder to at least decode image data received froman external source, and with the controller forwarding decoded imagedata from the decoder to the display panel and controlling the displaypanel to display the decoded image data in the second mode.
 18. Thedisplay apparatus of claim 17, wherein the controller selectivelycontrols opacities of pixels and/or sub-pixels of the display panel toproduce patterns on the display panel for displaying the image on thedisplay panel, in a second mode, and to form at least one pattern on thedisplay panel controlled to comprise at least one transparent pixel orsub-pixel and an area of the display panel surrounding the at least onepattern controlled to be opaque, to capture an image by at least onesensor unit of the sensor panel, in a first mode, wherein the displaypanel is controlled by the controller differently in the first mode andthe second mode.
 19. The display apparatus of claim 17, wherein thecontroller controls a monitoring of movement of an object, representedin the first incident light, over plural captured images andrespectively generated plural 3D information for the object anddetermines whether the monitored movement and generated plural 3Dinformation indicates that a gesture, of plural defined gestures, isbeing made by the movement of the object.
 20. The display apparatus ofclaim 1, wherein: the sensor panel comprises a first sensor unit and asecond sensor unit to sense the incident first light, and each of thefirst sensor unit and the second sensor unit comprise a unit colorfilter having a different color filter.
 21. The display apparatus ofclaim 20, wherein plural sensor units of the sensor panel arecollectively organized in different Bayer patterns.
 22. The displayapparatus of claim 1, wherein the sensor panel comprises a first sensorunit and a second sensor unit to sense the incident first light, andeach of the first sensor unit and the second sensor unit comprise aplurality of color filters respectively filtering different colors. 23.The display apparatus of claim 1, further comprising a display panel, asthe display, wherein, in a first mode, the controller controls thedisplay panel to selectively form at least one pattern to pass the firstincident light through the display panel to the sensor panel byselectively controlling respective opacities of pixels or sub-pixels ofthe display panel to be different, controlled differently than when thecontroller controls the display panel to display the image, in a secondmode.
 24. The display apparatus of claim 23, wherein the controllercontrols, at a current time in the first mode, the display panel to haveat least one pattern different than a pattern formed at a previous timewhen the display apparatus was in the first mode, by changing at leastone of a number of first pixels of the display panel that areselectively controlled to be opaque during the first mode, a number ofsecond pixels of the display panel that are selectively controlled to betransparent during the first mode, positions of one or more pixels ofthe display panel that are controlled to be opaque during the firstmode, and positions of one or more pixels of the display panel that arecontrolled to be transparent pixels during the first mode, than in theprevious time.
 25. The display apparatus of claim 23, wherein thecontroller selectively controls the display panel to form, as the atleast one pattern, at least one of a circular pattern, a polygonalpattern, and a modified uniformly redundant array (MURA) by controllingplural first pixels or plural first sub-pixels to be opaque and pluralsecond pixels or plural second sub-pixels to be transparent within apredetermined area of the display panel.
 26. The display apparatus ofclaim 25, wherein the predetermined area is one of 11×11 pixels, 11×11sub-pixels, 17×17 pixels, 17×17 sub-pixels, 23×23 pixels, and 23×23sub-pixels.
 27. The display apparatus of claim 1, wherein the sensorpanel is constructed of a substrate material that does not transmitlight.
 28. The display apparatus of claim 1, wherein the first pluraldistinct portions and the second plural distinct portions are arrangedin a grid pattern across the sensor panel.
 29. The display apparatus ofclaim 1, wherein the controller is further configured to encode thefirst incident light in a 3D graphics data format based on thedetermined depth.
 30. The display apparatus of claim 1, wherein thecontroller is further configured to restore an image of a view based onthe determined depth and the first incident light.
 31. A displayapparatus, comprising: a sensor panel configured to sense a firstincident light, incident to the sensor panel subsequent to passingthrough a display, and configured to at least partially blocktransmission of light through the sensor panel; and a controllerconfigured to selectively control the display to pass the first incidentlight toward the sensor panel, to control the display to pass light outof the display to display an image, and to determine a depth of anobject represented in the first incident light based upon at least onedetected size of a blur of at least one pattern detected by the sensorpanel, wherein the sensor panel comprises first plural distinct portionsconfigured to each forward light from the sensor panel towards thedisplay and second plural distinct portions configured to each senseincident light intensity, wherein the second plural distinct portionsare distinct from the first plural distinct portions and blocktransmission of light through the sensor panel toward the display, andwherein the controller restores the captured image by calculating adistance-based impulse response based on the determined depth and byperforming devolution of the captured image using the distance-basedimpulse response to compensate for refraction or geometric effectscaused by the at least one pattern of the display, wherein thedevolution is performed according to:I(x,y)=I′(x,y)*H ⁻¹(x,y), wherein I(x, y) represents the restored image,I′(x, y) represents the captured image with blurred image data, andH⁻¹(x, y) represents the distance-based impulse response.
 32. A displayapparatus, comprising: a sensor panel configured to sense a firstincident light, incident to the sensor panel subsequent to passingthrough a display, and configured to at least partially blocktransmission of light through the sensor panel; and a controllerconfigured to selectively control the display to pass the first incidentlight toward the sensor panel, to control the display to pass light outof the display to display an image, and to determine a depth of anobject represented in the first incident light based upon at least onedetected size of a blur of at least one pattern detected by the sensorpanel, wherein the sensor panel comprises first plural distinct portionsconfigured to each forward light from the sensor panel towards thedisplay and second plural distinct portions configured to each senseincident light intensity, wherein the second plural distinct portionsare distinct from the first plural distinct portions and blocktransmission of light through the sensor panel toward the display, andwherein the controller restores the captured image by calculating adistance-based impulse response based on a determined depth of anobject, represented in the first incident light, and by performingdevolution of a captured image using the distance-based impulse responseto compensate for refraction aid/or geometric effects caused by apattern of the display panel from where the first incident light passedthrough the display panel, wherein the determined depth of the object iscalculated by the controller based upon at least one detected size of ablur of the first incident light, as incident on the sensor panel,wherein the devolution is performed according to:I(x,y)=I′(x,y)*H ⁻¹(x,y), wherein I(x, y) represents the restored image,I′(x, y) represents the captured image with blurred image datacomprising the blur of the first incident light, and H⁻¹(x, y)represents the distance-based impulse response.
 33. A display apparatus,comprising: a sensor panel configured to sense a first incident light,incident to the sensor panel subsequent to passing through a display,and configured to at least partially block transmission of light throughthe sensor panel, wherein the sensor panel comprises plural distinctportions configured to each forward light from the sensor panel towardsthe display; a controller configured to selectively control the displayto pass the first incident light toward the sensor panel, to control thedisplay to pass light out of the display to display an image, todetermine a depth of an object represented in the first incident lightbased upon at least one detected size of a blur of at least one patterndetected by the sensor panel, and to restore a captured image bycalculating a distance-based impulse response based on the determineddepth by performing devolution of the captured image using thedistance-based impulse response to compensate for refraction orgeometric effects caused by the at least one pattern of the display; anda backlight unit configured to generate the light to be forwarded by theplural distinct portions of the sensor panel, wherein the sensor panelcomprises plural sensor units configured to sense the first incidentlight, in a first mode, and plural apertures, respectively as the pluraldistinct portions, configured to pass the second light emitted from thebacklight unit towards the display in first and second modes, with thecontroller controlling the display to display an image in the secondmode, wherein the plural sensor units and plural apertures are arrangedin a grid pattern across the sensor panel.
 34. The display apparatus ofclaim 33, wherein the apertures respectively comprise a diffuser todiffuse the second light.
 35. A display apparatus, comprising: a sensorpanel configured to sense a first incident light, incident to the sensorpanel subsequent to passing through a display, and configured to atleast partially block transmission of light through the sensor panel;and a controller configured to selectively control the display to passthe first incident light toward the sensor panel, to control the displayto pass light out of the display to display an image, to determine adepth of an object represented in the first incident light based upon atleast one detected size of a blur of at least one pattern detected bythe sensor panel, and to restore a captured image by calculating adistance-based impulse response based on the determined depth and byperforming devolution of the captured image using the distance-basedimpulse response to compensate for refraction or geometric effectscaused by the at least one pattern of the display, wherein the sensorpanel comprises plural distinct portions configured to each forwardlight from the sensor panel towards the display, and wherein thecontroller further comprises a sensor image restoring unit tointerpolate image data acquired by the sensing of the first incidentlight and to restore image data for light incident on non-sensingportions of the sensor panel that is not captured by any sensing of thesensor panel at the plural distinct portions of the sensor panel.
 36. Adisplay apparatus, comprising: a display panel; a sensor panelcomprising plural sensor units configured to sense incident light fromthe display panel and plural distinct portions configured to eachforward light from the sensor panel towards the display panel, theplural sensor units being distinct from the plural distinct portions;and a controller to selectively control opacities of pixels and/orsub-pixels of the display panel to produce patterns on the display panelfor displaying an image on the display panel, in a second mode, to format least one pattern on the display panel controlled to be transparentand an area of the display panel surrounding the at least one patterncontrolled to be opaque, to capture an image by the at least one sensorunit, in a first mode, to determine a depth of an object represented inthe first incident light based upon at least one detected size of a blurof the at least one pattern detected by the sensor panel, and to restorea captured image by calculating a distance-based impulse response basedon the determined depth by performing devolution of the captured imageusing the distance-based impulse response to compensate for refractionor geometric effects caused by the at least one pattern of the displaywherein the plural distinct portions are respectively plural aperturesor plural backlight units, with one or more of the plural aperturescomprising a diffuser element, and with each of the plural backlightunits respectively generating and directing light toward the displaypanel.
 37. The display apparatus of claim 36, wherein the plural sensorunits and plural distinct portions are arranged in a grid pattern acrossthe sensor panel.
 38. A display method, comprising: selectivelyconfiguring a display to form a pattern with at least one transparentpixel or sub-pixel to pass a first incident light through the displayand toward a sensor panel, behind the display, and to form an opaquearea that at least surrounds the pattern, when a current mode is a firstmode, and configuring the display to pass light from a backlight, behindthe display, through the display, when the current mode is a secondmode; sensing the first incident light, having passed through thedisplay, upon incidence to the sensor panel that at least partiallyblocks transmission of light, in a first mode; displaying an image onthe display by passing light in a direction from the sensor layerthrough the display; repeatedly controlling the current mode to only oneof the first mode and the second mode, comprising at least a changing ofthe current mode plural times; selectively controlling a combination ofa capture rate, for capturing images from the sensor panel, and adisplay rate, for displaying plural images, to be greater than a maximumrefresh rate of the display; determining a depth of an objectrepresented in the first incident light based upon at least one detectedsize of a blur of at least one pattern detected by the sensor panel;restoring a captured image by calculating a distance-based impulseresponse based on the determined depth; and performing devolution of thecaptured image using the distance-based impulse response to compensatefor refraction and/or geometric effects caused by the at least onepattern of the display.
 39. The method of claim 38, further comprisingcontrolling, at a current time in the first mode, the display to have atleast one pattern different than a pattern formed at a previous timewhen the current mode was the first mode, by changing at least one of anumber of first pixels of the display that are selectively controlled tobe opaque during the first mode, a number of second pixels of thedisplay that are selectively controlled to be transparent during thefirst mode, positions of one or more pixels of the display that arecontrolled to be opaque during the first mode, and positions of one ormore pixels of the display that are controlled to be transparent pixelsduring the first mode, than in the previous time.
 40. The method ofclaim 38, further comprising controlling, at a current time in the firstmode, the display to have at least one pattern different than a patternformed at a previous time when the current mode was the first mode, bychanging at least one of a number of first sub-pixels of the displaythat are selectively controlled to be opaque during the first mode, anumber of second sub-pixels of the display that are selectivelycontrolled to be transparent during the first mode, positions of one ormore sub-pixels of the display that are controlled to be opaque duringthe first mode, and positions of one or more sub-pixels of the displaythat are controlled to be transparent pixels during the first mode, thanin the previous time.
 41. The method of claim 38, further comprisingselectively controlling the display to only pass the first incidentlight toward the sensor panel in a first cycle, of plural fixed timelength cycles, and controlling the display to only pass the light out ofthe display to display the image in a second cycle, of the plural fixedtime length cycles, wherein the fixed time is a time it takes for thedisplay to perform one refresh.
 42. The method of claim 38, furthercomprising selectively controlling opacities of pixels and/or sub-pixelsof the display to produce patterns on the display for displaying theimage on the display, in a second mode, and to form at least one patternon the display controlled to be transparent and an area of the displaysurrounding the at least one pattern controlled to be opaque, to capturean image by at least one sensor unit of the sensor, in a first mode,wherein the display is controlled differently in the first mode and thesecond mode.
 43. The method of claim 38, wherein the determining furthercomprises determining plural depth measurements of the objectrepresented in plural first incident lights each sensed by the sensorpanel based respectively upon at least one detected size of a blurpattern for each sensed incident light, and generating 3D information ofthe object.
 44. The method of claim 43, further comprising monitoring amovement by the object over plural captured images and respectivelygenerated plural 3D information and determining whether the monitoredmovement and generated plural 3D information indicates that a gesture,of plural defined gestures, is being made by the movement of the object,and controlling an operation of a device based on the determinedgesture.
 45. A display method, comprising: selectively configuring adisplay to form a pattern with at least one transparent pixel orsub-pixel to pass a first incident light through the display and towarda sensor panel, behind the display, and to form an opaque area that atleast surrounds the pattern, when a current mode is a first mode, andconfiguring the display to pass light from a backlight, behind thedisplay, through the display, when the current mode is a second mode,sensing the first incident light, having passed through the display,upon incidence to the sensor panel that at least partially blockstransmission of light, in a first mode; displaying an image on thedisplay by passing light in a direction from the sensor layer throughthe display, repeatedly controlling the current mode to only one of thefirst mode and the second mode, comprising at least a changing of thecurrent mode plural times; selectively controlling a combination of acapture rate, for capturing images from the sensor panel, and a displayrate, for displaying plural images, to be greater than a maximum refreshrate of the display; determining a depth of an object represented in thefirst incident light based upon at least one detected size of a blur ofat least one pattern detected by the sensor panel; restoring a capturedimage by calculating a distance-based impulse response based on thedetermined depth; performing devolution of the captured image using thedistance-based impulse response to compensate for refraction orgeometric effects caused by the at least one pattern of the display; andencoding the restored captured image, wherein the devolution isperformed according to:I(x,y)=I′(x,y)*H ⁻¹(x,y), wherein I(x, y) represents the restored image,I′(x, y) represents the captured image with blurred image data, andH⁻¹(x, y) represents the distance-based impulse response.
 46. A displaymethod, comprising: selectively configuring a display to form a patternwith at least one transparent pixel or sub-pixel to pass a firstincident light through the display and toward a sensor panel, behind thedisplay, and to form an opaque area that at least surrounds the pattern,when a current mode is a first mode, and configuring the display to passlight from plural backlights of the sensor panel through the display,when the current mode is a second mode; sensing the first incidentlight, having passed through the display, upon incidence to the sensorpanel that blocks transmission of light, in a first mode; displaying animage on the display by passing respectively generated light from eachbacklight from the sensor layer through the display; repeatedlycontrolling the current mode to be one of the first mode and the secondmode, comprising at least a changing of the current mode plural times;selectively controlling a combination of a capture rate, for capturingimages from the sensor panel, and a display rate, for displaying pluralimages, to be greater than a maximum refresh rate of the display;determining a depth of an object represented in the first incident lightbased upon at least one detected size of a blur of the at least onepattern detected by the sensor panel; restoring a captured image bycalculating a distance-based impulse response based on the determineddepth; and performing devolution of the captured image using thedistance-based impulse response to compensate for refraction and/orgeometric effects caused by the at least one pattern of the display. 47.A display apparatus, comprising: a sensor panel configured to sense afirst incident light, incident to the sensor panel subsequent to passingthrough a display, and configured to at least partially blocktransmission of light through the sensor panel; and a controllerconfigured to selectively control the display to pass the first incidentlight toward the sensor panel, to control the display to pass light outof the display to display an image, and to determine a depth of anobject represented in the first incident light based upon at least onedetected size of a blur of at least one pattern detected by the sensorpanel, wherein the sensor panel comprises first plural distinct portionsconfigured to each forward light from the sensor panel towards thedisplay and second plural distinct portions configured to each senseincident light intensity, wherein the second plural distinct portionsare distinct from the first plural distinct portions and blocktransmission of light through the sensor panel toward the display, andwherein the controller uses respective captured image data of an objectfrom two or more second plural distinct portions to perform a devolutionof a captured image of the object made up of the respective captureddata, using a distance-based impulse response to compensate forrefraction or geometric effects caused by at least one pattern of thedisplay, to restore the captured image of the object, wherein thedevolution is performed according to:I(x,y)=I′(x,y)*H ⁻¹(x,y), wherein I(x, y) represents the restored image,I′(x, y) represents the captured image with blurred image data, andH⁻¹(x, y) represents the distance-based impulse response.
 48. Thedisplay apparatus of claim 47, wherein the controller determines pluraldepth measurements of the object represented in plural first incidentlights each sensed by the two or more of the second plural distinctportions based respectively upon at least one detected size of a blurpattern for each sensed incident light, and wherein the distance-basedimpulse response is based upon the determined plural depth measurements.49. The display apparatus of claim 47, wherein the controller furthercomprises a sensor image restoring unit to interpolate image dataacquired by the sensing of the first incident light and to restore imagedata that is not captured by any sensing of the sensor panel at theplural distinct portions of the sensor panel, in the restoration of thecaptured image of the object.